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INTRODUCTION 


Achieving  information  dominance  (ID)  in  the  military  battlespace  has  become  a  major 
operational  thrust.  Defined  as  "the  ability  to  collect,  control,  exploit,  and  defend 
information  while  denying  an  adversary  the  ability  to  do  the  same"  (Department  of  the  Air 
Force,  1995),  ID  is  seen  as  the  key  controlling  the  battle  and  winning  future  armed 
conflicts.  "Dominating  the  information  spectrum  is  as  critical  to  conflict  now  as  occupying 
the  land  or  controlling  the  air  has  been  in  the  past."  (General  Ronald  R.  Fogleman 
(Department  of  the  Air  Force,  1995)). 

The  concept  of  ID  and  the  issues  involved  in  achieving  it  have  been  discussed  based  on  a 
model  of  situation  awareness  (SA)  within  the  context  of  complex,  distributed  crews  (or 
military  units)  as  is  envisioned  in  future  military  operations  (Endsley  &  Jones,  1997).  By 
examining  what  is  known  about  how  people  access,  assimilate,  and  interpret  information  to 
develop  SA  and  how  this  fits  within  the  decision  making  and  action  cycle,  directions  for 
the  development  of  systems  to  support  the  goal  of  ID  were  established.  Achieving  ID 
involves  far  more  than  having  more  data  than  the  enemy.  It  requires  that  the  data  be 
transformed  into  the  required  information  in  a  timely  manner  for  a  multitude  of  forces, 
each  with  varied  and  dynamically  changing  but  inter-related  information  needs,  and 
properly  understood  by  each  within  the  context  of  a  joint  mission.  Fulfilling  this 
requirement  requires  an  understanding  of  SA  and  the  factors  that  impact  it  for  each  of  the 
distributed  teams  involved,  and  the  way  in  which  these  S  A  needs  interact  between  teams. 

In  military  operations,  most  actions  occur  in  teams  or  crews  of  individuals.  Examining  SA 
as  it  exists  within  teams  and  between  teams  that  are  involved  in  achieving  a  common  goal 
lends  an  important  perspective  for  the  determination  of  system  designs  that  support  the 
complex  inter-related  activities  of  these  teams.  Team  SA  has  been  defined  as  "the  degree 
to  which  every  team  member  possess  the  SA  required  for  his  or  her  responsibilities" 
(Endsley,  1995).  The  degree  to  which  team  members  possess  a  shared  understanding  of  the 
situation  with  regard  to  their  shared  SA  requirements  is  an  extremely  important  aspect  of 
team  SA.  Shared  SA  can  be  defined  as  "the  degree  to  which  team  members  possess  the 
same  SA  on  shared  SA  requirements."  Shared  SA  can  be  depicted  as  the  shaded  area  in 
Figure  1,  where  each  circle  represents  the  SA  requirements  of  each  team  member.  It  is  the 
area  where  these  requirements  overlap  that  constitutes  the  need  for  a  shared  understanding 
of  the  situation  within  a  team.  Similarly,  where  the  SA  requirements  overlap  between 
teams,  a  shared  understanding  of  this  information  is  equally  important  for  the  ability  of  the 
teams  to  achieve  their  goals. 

Developing  shared  SA  within  a  team  and  between  teams  can  be  extremely  challenging, 
especially  where  those  teams  are  distributed  in  terms  of  space,  time,  or  physical  barriers. 
This  has  been  described  in  the  model  of  team  SA  as  a  function  of  four  components 
(Endsley  &  Jones,  1997): 
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(1)  Shared  SA  Requirements  -  the  degree  to  which  the  team  members  know  which 
information  needs  to  be  shared,  including  their  higher  level  assessments  and  projections 
(which  are  usually  not  otherwise  available  to  fellow  team  members),  and  information  on 
team  members’  task  status  and  current  capabilities. 


Shared  SA 
Requirements 


Figure  1.  Shared  SA  Requirements 


(2)  Shared  SA  Devices  -  the  devices  available  for  sharing  this  information,  which  can 
include  direct  communication  (both  verbal  and  non-verbal).  Shared  Displays  or  a  shared 
environment.  As  non-verbal  communication  and  a  shared  environment  are  usually  not 
available  in  distributed  teams,  this  places  far  more  emphasis  on  verbal  communication  and 
technologies  for  creating  shared  information  displays. 

(3)  Shared  SA  Mechanisms  -  the  degree  to  which  team  members  possess  mechanisms,  such 
as  Shared  Mental  Models,  which  support  their  ability  to  interpret  information  in  the  same 
way  and  make  accurate  projections  regarding  each  other’s  actions.  The  possession  of 
Shared  Mental  Models  can  greatly  facilitate  communication  and  coordination  in  team 
settings. 

(4)  Shared  SA  Processes  -  the  degree  to  which  team  members  engage  in  effective  processes 
for  sharing  SA  information  which  may  include  a  group  norm  of  questioning  assumptions, 
checking  each  other  for  conflicting  information  or  perceptions,  setting  up  coordination  and 
prioritization  of  tasks,  and  establishing  contingency  planning  among  others. 
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Objective 


The  objective  of  this  study  was  to  experimentally  test  this  model  of  SA  using  a  simulation 
test-bed  that  incorporates  features  of  a  distributed  team  architecture,  as  would  be  found  in  a 
military  battlespace.  In  particular,  this  effort  focused  on  examining  the  use  of  shared 
battlespace  displays  and  the  role  of  shared  mental  models.  While  it  has  been  hypothesized 
that  shared  displays  and  shared  mental  models  would  assist  team  members  in  performing 
joint  tasks,  this  hypothesis  has  never  been  empirically  evaluated. 
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METHOD 


Participants 

Thirty  four  participants  served  as  paid  subjects  in  this  research.  Two  participants  (one 
team)  were  dropped  from  the  analysis  due  to  their  low  overall  performance  (less  than  three 
standard  deviations  below  the  mean  on  performance  measures).  The  remaining  32 
participants  were  tested  in  pairs  for  a  total  of  16  teams.  The  participants  (mean  age  =  21.72 
years;  range  18  -  36  years)  comprised  of  10  men  and  22  women  with  an  average  of  14.9 
years  of  formal  education.  All  but  one  participant  indicated  they  had  normal  or  better 
vision.  The  participants  had  an  average  of  6.8  years  of  computer  experience  (range  2  -  22), 
so  all  were  familiar  with  the  general  operation  of  a  personnel  computer. 

Design 

Two  factors  served  as  independent  variables  in  the  study:  Shared  Mental  Models  and 
Shared  Displays.  Presence  or  absence  of  a  Shared  Mental  Model  was  a  between  team 
manipulation,  while  presence  or  absence  of  a  Shared  Display  was  a  within  team 
manipulation.  The  use  of  Shared  Displays  was  counterbalanced  across  teams.  Therefore, 
half  of  the  teams  participated  in  the  task  with  Shared  Displays  first,  followed  by  Non- 
shared  Displays.  The  other  half  of  the  teams  begun  the  task  without  Shared  Displays, 
followed  by  Shared  Displays,  as  shown  in  Table  1.  Each  team  completed  five  ten-minute 
blocks  in  each  of  the  two  display  conditions. 

Table  1.  Experimental  Design 


Shared  Mental  Model 

Non-shared  Mental  Model 

Shared  Display/ 

Team  1 

Team  2 

Non>shared  Display 

Teams 

Team  6 

Team  9 

Team  10 

Team  17 

Team  14 

Non-shared  Display/ 

Team  3 

Team  4 

Shared  Display 

Team  7 

Team  8 

Team  11 

Team  12 

Team  15 

Team  16 

The  effects  of  the  independent  variables  on  the  performance  of  the  two  decision  makers  in 
the  Theatre  Defense  task  were  examined  as  dependent  variables.  The  processing  outcome 
(destroyed,  passed  through,  or  collided),  time  to  process  a  target,  and  reward  and  penalty 
points  were  recorded  for  the  Air  Commander.  The  time  to  make  an  identification, 
correctness  of  identification,  and  use  of  the  Air  Commander’s  prioritization  order  were 
recorded  for  the  Intelligence  Officer.  These  variables  were  also  used  to  examine  the  effects 
of  information  dissonance  and  missing  information  on  decision  making  under  time 
pressure. 
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Procedure 


Teams  were  tested  one  at  a  time.  Prior  to  performing  the  Theatre  Defense  task,  each  team 
member  was  asked  to  fill  out  an  informed  consent  sheet  as  well  as  a  short  background 
questionnaire  that  asked  about  formal  schooling,  age,  and  computer  use.  After  completion 
of  the  questionnaire,  team  members  were  given  a  handout  describing  their  task  within 
Theatre  Defense  (see  Appendix  A  for  the  Intelligence  Officer’s  job  description  and 
Appendix  B  for  the  Air  Commander’s  job  description). 

Teams  in  the  Shared  Mental  Model  condition  were  then  asked  to  read  each  other’s  job 
description  in  order  to  formulate  a  Shared  Mental  Model  of  the  joint  task.  After  reading 
the  instructions,  they  were  given  time  to  ask  questions,  discuss,  and  formulate  a  joint 
strategy  to  optimize  their  performance.  In  the  Non-shared  Mental  Model  condition, 
participants  were  given  time  to  ask  questions,  however,  the  other  team  member  was  not  in 
the  room  during  this  period.  In  this  condition,  they  were  only  given  information  regarding 
their  own  task  and  no  discussion  was  allowed  between  the  team  members. 

In  the  Shared  Display  manipulation,  the  two  team  members  were  seated  side  by  side  with 
their  computer  monitors  approximately  six  inches  apart.  This  placement  allowed  them  the 
ability  to  view  the  other  team  member’s  computer  screen  while  performing  the  task.  In  the 
Non-shared  Display  manipulation,  participants  were  also  seated  side  by  side,  but  they  were 
separated  by  a  barrier  and  thus  could  not  view  each  other  or  the  other  team  member’s 
displays. 

Teams  completed  two  ten-minute  practice  trials.  Questions  were  answered  at  the 
completion  of  each  trial.  In  the  Shared  Mental  Model  condition,  the  team  could  work  on  a 
joint  strategy  both  between  practice  trials  and  at  the  conclusion  of  the  practice.  The  test 
consisted  of  five  ten-minute  trials  in  the  Non-shared  Display  condition  and  five  in  the 
Shared  Display  condition.  Participants  were  given  a  15  minute  break  after  each  five  trial 
block.  In  the  Shared  Mental  Model  condition,  participants  were  allowed  to  discuss  the  task 
during  the  break,  but  not  between  the  trials. 

Task 

A  new  task  paradigm  was  created  for  exploring  decision  making  and  SA  issues  in  a  team 
task  appropriate  for  the  information  warfare  environment.  The  microworld,  entitled  Theatre 
Defense,  incorporates  activities  by  two  individuals:  an  Intelligence  Officer  and  an  Air 
Commander  who  each  have  separate,  but  inter-related  tasks.  The  two  team  members  work 
at  separate  workstations,  connected  by  an  ethemet  LAN.  The  role  of  the  Air  Commander 
is  to  protect  the  home  base  from  incoming  aircraft.  Targets  (designated  by  blank  boxes) 
appear  on  the  radar  screen  moving  towards  a  central  point  (the  home  base),  as  shown  in 
Figure  2.  The  Air  Commander  must  prioritize  these  targets  (based  on  range  and  speed)  and 
request  information  from  the  Intelligence  Officer  on  their  identity  and  mission  priority. 
Once  an  identification  has  been  received  from  the  Intelligence  Officer,  the  Air  Commander 
processes  the  targets  accordingly.  The  Air  Commander  must  choose  which  targets  to 
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destroy  (based  on  range,  speed,  and  penalty/reward  points)  and  which  to  let  through  to  the 
home  base  (such  as  friendlies). 

Targets  included  fighters,  bombers,  and  transport  aircraft  of  either  friendly  or  enemy 
designations,  making  for  a  possibility  of  six  categories  with  a  total  of  18  different  aircraft 
types.  Points  were  assigned  to  each  category  representing  the  reward  points  for  destroying 
the  aircraft  and  penalty  points  for  allowing  the  aircraft  to  get  through  to  the  home  base. 
Reward  and  penalty  points  for  each  category  were  based  on  the  mission  relevance  and 
lethality  of  the  aircraft  type.  Friendly  aircraft  had  a  zero  penalty  for  getting  through  to 
home  base  and  a  negative  reward  associated  with  destroying  them.  In  addition  to  landing 
or  destroying  an  aircraft,  it  was  possible  for  some  targets  to  collide  with  each  other, 
resulting  in  the  reward  and  penalty  points  associated  with  both  aircraft  to  be  recorded. 
Thus,  the  Air  Commander  needed  to  correctly  destroy  enemy  aircraft  that  would  conflict 
with  friendly  aircraft  before  such  a  collision  occurred. 


Figure  2.  Air  Commander  Workstation’s  Screen 
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The  Intelligence  Officer  was  supplied  with  a  list  of  targets  and  the  identifications  provided 
by  several  different  sensors  of  varying  reliability,  as  shown  in  Figure  3.  Upon  a  request 
from  the  Air  Commander,  the  Intelligence  Officer  needed  to  select  the  sensor  information 
for  that  target  and  make  a  designation  of  the  target  aircraft  type  and  category.  Three 
sensors  (A,  B,  and  C)  were  provided  which  participants  were  instructed  had  reliability  rates 
of  75%,  50%,  and  50%,  respectively.  To  represent  issues  present  in  the  real  world,  the 
information  from  the  three  sensors  was  either  consistent  (1/3  of  the  cases),  partially 
missing  (1/3  of  the  cases),  or  dissonant  (1/3  of  the  cases).  As  decision  making  in  the  face 
of  information  dissonance  was  of  particular  concern,  conflicting  (dissonant)  sensor 
information  was  provided  for  1/3  of  the  targets  where  different  sensors  indicated  the  target 
was  of  the  same  category  but  of  two  different  types  (e.g.,  F18  vs  F16)  or  of  two  different 
categories  (e.g.,  FI 6  vs  SU35).  The  distribution  of  the  dissonant  information  was 
distributed  across  sensor  types  and  reliabilities  in  order  to  examine  these  effects.  For 
instance,  in  some  cases  two  sensors  indicated  a  friendly  and  one  an  enemy,  and  in  other 
cases  two  indicated  an  enemy  and  one  a  friendly.  Which  sensor  provided  the  dissonant 
information  was  also  counter-balanced  across  cases  to  provide  for  information  on  the  effect 
of  sensor  reliability  as  well  as  sensor  confirmation  in  making  identification  decisions. 
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Figure  3.  Intelligence  Officer’s  Workstation  Screen 
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Once  the  Intelligence  Officer  made  a  decision,  this  information  was  passed  to  the  Air 
Commander  who  saw  the  target  change  colors  (indicating  its  category)  and  the  reward  and 
penalty  points  associated  with  the  target.  This  information  was  dependent  on  the 
Intelligence  Officer’s  identification  and  therefore  may  or  may  not  have  been  correct. 
However,  the  points  assigned  when  the  target  was  processed  (landed,  destroyed  or 
collided)  were  based  on  the  actual  identification  of  the  target.  Feedback  was  provided  only 
after  the  target  was  destroyed  or  allowed  to  pass  through  to  the  home  base  (as  in  the  real 
world)  to  the  Intelligence  Officer  (e.g.  F-16  passed  through;  F-16  destroyed).  The  Air 
Commander  was  also  provided  with  the  ranning  point  total.  Learning  therefore  was  able 
to  take  place  allowing  the  team  members  to  develop  effective  strategies  and  decision 
behaviors. 

The  pace  of  the  task  was  such  that  to  maximize  points,  both  officers  needed  to  be  very 
strategic  about  how  they  prioritized  and  processed  targets.  Otherwise  a  significant  number 
of  enemy  aircraft  would  penetrate  the  air  defense  and  strike  the  home  base  or  a  significant 
number  of  friendly  aircraft  would  fall  prey  to  fratricide. 

In  the  Non-Shared  Display  condition,  each  officer  had  only  a  limited  amount  of 
information  from  the  other,  in  addition  to  their  own  display.  The  Intelligence  Officer  saw 
only  the  prioritization  list  provided  by  the  Air  Commander  and  the  Air  Commander  saw 
only  the  resultant  classification  provided  by  the  Intelligence  Officer. 

In  the  Shared  Display  condition,  each  officer  also  saw  the  “big  picture,”  showing  all  the 
display  information  of  the  other  officer.  Therefore  the  Intelligence  Officer  also  saw  a 
picture  of  the  Air  Commander’s  radar  display  which  it  was  hypothesized  would  help 
him/her  better  prioritize  and  anticipate  the  prioritization  of  the  targets  to  be  identified.  The 
Air  Commander  also  saw  the  underlying  sensor  data  generating  the  target  identification. 
This  was  hypothesized  to  lead  to  a  benefit  in  correct  processing  of  targets.  On  the  other 
hand,  the  extra  information  provided  may  lead  to  overload  and  slow  down  decision 
making. 

The  Shared  Mental  Model  condition  was  investigated  by  manipulating  the  instructions  that 
were  provided  to  each  officer.  In  the  Non-Shared  Mental  Model  condition,  each  team 
member  was  provided  with  only  the  instractions  for  their  part  of  the  task.  In  the  Shared 
Mental  Model  condition,  each  was  also  provided  with  the  instructions  for  the  other  team 
member. 

The  Theatre  Defense  program  was  written  in  Microsoft  Visual  Basic.  It  was  based  on 
Multitask,  a  single  person  control  task  created  by  Kaber  and  Endsley  (1997).  Theatre 
Defense  was  hosted  on  two  separate  Pentium  based  workstations  that  were  connected  by  an 
Ethernet  LAN.  Data  for  each  team  member  was  recorded  on  their  workstation  computer. 
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RESULTS 


Scoring 

Data  was  collected  from  both  participants  during  the  trials.  In  order  to  facilitate  data 
analysis  (as  each  team  produced  more  than  3,000  separate  target  entries),  summary  files 
were  created  containing  means  for  the  variables  of  interest  for  each  of  the  10  trial  blocks. 
Mean  penalty  points,  mean  reward  points,  and  mean  decision  time  to  expiration,  collision, 
or  attack  were  calculated  for  each  block  for  each  team.  Mean  time  from  target  information 
request  to  classification,  mean  target  viewing  time,  the  percentage  of  targets  that  were 
requested  by  the  Air  Commander  at  time  of  classification,  and  the  percentage  of  targets 
that  were  at  the  top  of  the  Air  Commander’s  request  list  at  the  time  of  classification  were 
also  calculated  for  each  block  for  each  team. 

Three  sets  of  analysis  were  conducted.  The  first  examined  the  effects  of  Shared  Mental 
Models  and  Shared  Displays  on  team  performance.  The  second  analysis  examined  the 
influence  of  Sensor  Performance  on  decision  making,  and  the  final  analysis  focused  in 
more  detail  on  the  effects  of  the  dissonant  data.  All  analyses  were  performed  using 
analysis  of  variance  (ANOVA).  Tukey  tests  were  used  for  post-hoc  analysis.  We  used  an 
alpha  level  of  .05  for  all  analyses. 


Mental  Models  and  Shared  Displays 

Penalty  points,  reward  points,  decision  making  time,  percentage  of  targets  classified  from 
the  request  list,  and  percentage  of  targets  classified  from  the  top  of  the  Air  Commander’s 
request  list  were  examined  using  a  2  by  4  by  5  (Shared  Mental  Model  by  Shared 
Display/Order  by  Block)  ANOVA.  Main  effects  and  two-way  interactions  were  included 
in  the  model.  (As  the  order  of  receiving  Shared  Displays  may  have  had  an  affect  as  well  as 
the  presence  or  absence  of  the  Shared  Displays,  this  variable  was  treated  as  having  four 
levels  in  the  analysis:  Shared  Display  first,  Non-Shared  Display  second,  Non-shared 
Display  first  and  Shared  Display  second.) 

Penalty  Points  and  Reward  Points 

Reward  points  did  not  significantly  vary  across  the  Shared  Mental  Model  and  Shared 
Display  conditions.  Mean  penalty  points  per  block  were  observed  to  be  higher  in  the  Non- 
Shared  Mental  Model  condition  than  in  the  Shared  Mental  Model  condition,  F(l,4)  = 
13,817,  p  =  .021,  as  shown  in  Figure  4.  Display  type  also  effected  the  penalty  points, 
F(3,12)  =  6.14,  p  =  .009,  as  shown  in  Figure  5.  Teams  that  started  out  without  Shared 
Displays  accraed  the  most  penalty  points.  Teams  that  were  provided  with  Shared  Displays 
and  teams  that  had  Non-Shared  Displays  after  a  period  in  which  they  had  been  provided 
with  Shared  Displays  performed  better.  Therefore,  one  interpretation  of  this  data  is  that  the 
Shared  Displays  were  useful  by  helping  to  build  up  a  mental  model  (thus  providing 
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Points  Points 


residual  effects  when  the  Shared  Displays  were  removed).  There  were  no  significant  main 
effects  of  Block  or  two-way  interactions. 


Shared  Non-shared 

Mental  Mental 

Model  Model 

Figure  4:  Mean  Penalty  Points  by  Shared  Mental  Model  Condition 
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Figure  5:  Mean  Penalty  Points  by  Shared  Display  Condition 
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Decision  Time 


The  time  data  present  an  interesting  picture.  The  participants  worked  together  as  a  team  on 
a  time  constrained  task.  Therefore,  any  time  the  Intelligence  Officer  was  slower  in  making 
identifications  limited  the  amount  of  time  the  Air  Commander  had  for  making  targeting 
decisions.  In  the  Non-Shared  Mental  Model  condition,  the  Intelligence  Officers  were 
significantly  slower  at  categorizing  information  than  those  in  the  Shared  Mental  Model 
condition,  F(l,4)  =  54.43,  p  =  .002.  This  left  the  Air  Commander  with  less  time  to  attack 
the  targets,  F(l,4)  =  90.73,  p  =  .001.  Mean  time  from  classification  to  landing  was 
similarly  reduced,  F(l,4)  =  10.49,  p  =  .032.  This  is  shown  in  Figure  6.  The  presence  of 
Shared  Mental  Models,  therefore,  was  shown  to  aid  classification  performance,  leaving 
more  time  for  the  attack  decision. 

Display  type  also  effected  decision  time  performance.  The  Intelligence  Officers  were 
slower  at  classifying  aircraft,  F(3,12)  =  10.36,  p  =  .001,  when  given  a  Shared  Display  than 
when  they  had  the  Non-shared  Display  condition,  as  shown  in  Figure  7.  They  were  the 
fastest  when  they  did  not  have  the  Shared  Display  after  having  had  the  Shared  Display 
first.  While  having  the  Shared  Display  slowed  them  down  slightly,  it  also  allowed  them  to 
build  up  a  mental  model  that  helped  them  after  the  Shared  Display  was  removed. 

Correspondingly,  the  Air  Commanders  had  less  time  to  process  aircraft,  with  mean  time  to 
landing  greatest  when  they  did  not  have  the  Shared  Display  after  first  having  the  Shared 
Display  condition,  F(3,12)  =  49.627,  p  <  .001.  This  is  also  the  condition  where 
classification  time  was  fastest.  The  Air  Commanders  took  significantly  less  time  to  attack 
incoming  planes,  F(3,12)  =  13.392,  p  <  .001,  in  the  Shared  Display  condition  when  it  came 
after  a  Non-shared  Display  condition,  showing  a  learning  effect. 
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Figure  6:  Decision  Time  by  Shared  Mental  Model  Condition 
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Figure  7:  Mean  Decision  Time  by  Shared  Display  Condition 
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The  interaction  of  Shared  Mental  Model  and  Shared  Display  was  significant  for  both  the 
time  it  took  the  Intelligence  Officer  to  classify  requested  targets,  F(3,  128)  =  3.103,  p  = 
.029,  shown  in  Figure  8,  and  the  time  it  took  the  Air  Commander  to  attack,  F(3,  130)  = 
2.825,  /?  =  .041,  shown  in  Figure  9.  As  shown  in  Figure  8,  the  Intelligence  Officer  was 
faster  at  classifying  targets  when  provided  a  Shared  Mental  Model  of  the  Air  Commander’s 
task  and  the  Non-shared  Display  condition.  When  they  did  not  have  a  Shared  Mental 
Model,  classification  time  was  shortest  in  the  Non-shared  Display  condition  only  if  it 
followed  the  Shared  Display  condition  which  compensated  for  the  lack  of  a  Shared  Mental 
Model. 

On  the  other  hand,  the  opposite  effect  was  observed  for  the  Air  Commander.  The  Air 
Commander  had  (and  used)  more  time  to  process  and  attack  incoming  planes  when  the 
team  was  given  the  Shared  Mental  Model  condition,  shown  in  Figure  9.  The  teams  that 
were  given  a  Shared  Mental  Model  but  not  a  Shared  Display  first  had  the  fastest 
classification  times  (as  the  Intelligence  Officer  was  not  slowed  down  by  looking  at  the  Air 
Commander’s  screen)  and  thus  the  longest  times  for  processing  and  attacking  by  the  Air 
Commander.  When  the  teams  had  Non-shared  Mental  Models  and  Non-shared  Displays 
first,  the  Intelligence  Officer  took  significantly  longer  to  send  over  the  requested  target 
information,  thus  leaving  the  Air  Commander  with  less  time  to  make  a  decision.  It 
appears  from  this  interaction,  that  either  the  presence  of  a  Shared  Mental  Model  or  Shared 
Display  first  will  lead  to  faster  decision  making  times. 
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Figure  8:  Mean  Classification  Time  for  Shared  Display  by  Shared 

Mental  Model  Interaction 
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Figure  9:  Mean  Time  to  Attack  for  Shared  Display  by  Shared 

Mental  Model  Interaction 


Classification  of  Priority  Targets 

The  degree  to  which  the  Intelligence  Officer  classified  targets  that  the  Air  Commander  had 
requested  was  examined  as  an  index  of  the  degree  to  which  the  two  participants  were 
operating  as  a  team.  The  percentage  of  targets  classified  that  were  on  the  request  list  was 
significantly  effected  by  Shared  Displays,  F(3,12)  =  135.289,  p  <  .001.  The  interaction  of 
Shared  Mental  Model  and  Shared  Display  was  also  significant  for  this  variable,  F(3,132)  = 
3.75,  p  =  .013.  This  effect  is  shown  in  Figure  10. 

Similarly,  the  percentage  of  targets  classified  that  were  at  the  top  of  the  request  list  (the 
highest  priority)  was  significantly  effected  by  Shared  Mental  Models,  F(l,4)  =  199.288,  p 
<  .001,  and  Shared  Displays,  F(3,12)  =  32.613,  p  <  .001.  There  was  also  a  significant 
Shared  Mental  Model  by  Shared  Display  interaction  for  this  variable,  F(3,132)  =  5.90,  p  = 
.001,  shown  in  Figure  11.  When  Air  Commanders  were  given  a  Shared  Display  first  and 
no  mental  model,  they  adopted  a  strategy  to  request  fewer  targets  during  the  final  five  trials 
(see  Figure  10).  The  Intelligence  Officer  also  sent  over  fewer  of  the  requested  top  priority 
aircraft  classifications  (see  Figure  1 1)  when  first  given  a  Shared  Display.  However,  when 
the  teams  were  not  given  a  Shared  Display  first,  and  had  no  Mental  Model,  they  requested 
more  aircraft  and  classified  more  of  the  high  priority  aircraft.  This  may  be  in  part  to  their 
inability  to  adopt  a  mental  model  of  the  joint  task  and  thus  the  development  of  a  strategy  to 
lower  the  workload  of  the  other  team  member. 
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;ure  10:  Percent  of  Targets  Classified  that  were  Requested  by  the  Air  Commander 
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Figure  11:  Percent  of  Targets  ClassiHed  from  the  Top  of  the  Request  List 


Sensor  Performance 


The  second  analysis  examined  the  effect  of  the  differing  Sensor  Performances  (all  three 
sensors  showing  the  same  information,  missing  sensor  information,  or  dissonant  sensor 
information)  on  the  performance  of  the  Intelligence  Officer.  During  the  study,  the 
Intelligence  Officer  would  view  the  available  sensor  information  to  determine  each 
aircraft’s  classification.  Four  Sensor  Patterns  were  presented  for  missing  and  dissonant 
Sensor  Performances:  Sensor  A  missing  or  dissonant,  sensor  B  missing  or  dissonant, 
sensor  C  missing  or  dissonant,  and  sensors  B  and  C  missing  or  dissonant  from  sensor  A.  A 
3  by  4  by  16  (Sensor  Performance  by  Sensor  Pattern  by  Team)  ANOVA  was  performed  on 
both  the  mean  viewing  time  and  correctness  of  classification  by  the  Intelligence  Officer. 

Mean  viewing  time  was  not  significantly  effected  by  any  of  the  variables  of  interest.  It 
appears  that  teams  did  not  take  longer  to  make  a  classification  decision  based  on  the 
differing  Sensor  Performance  types  or  Sensor  Patterns.  This  may  be  in  part  due  to  the 
speeded  nature  of  the  task  which  pushes  the  Intelligence  Officer  to  classify  as  quickly  as 
possible. 

There  was  a  significant  main  effect  of  Team,  Sensor  Performance,  and  Sensor  Pattern  on 
the  correctness  of  the  classification,  however.  Two  teams  were  lower  in  the  percentage  of 
correct  classifications,  as  compared  to  the  remaining  14  teams,  F(15,90)  =  7.76,  p  <  .001. 
However,  these  teams  did  surprisingly  well  in  terms  of  overall  points  and  speed. 

Teams  incorrectly  classified  targets  with  dissonant  sensor  data  more  frequently  than  those 
with  full  or  missing  sensor  data,  F(2,  30)  =  241.905,  p  <  .001.  Surprisingly,  missing  sensor 
data  was  not  a  problem  for  classification,  although  a  sensor  that  showed  dissonant  data 
was.  The  pattern  of  the  dissonant  or  missing  sensor  also  had  an  effect,  F(3,45)  =  15.47,  p  < 
.001.  When  dissonant  or  missing  data  was  presented  on  sensor  A  (the  first  and  most 
reliable  of  the  three)  or  on  sensors  B  and  C,  the  Intelligence  Officer  incorrectly  classified 
the  targets  more  often. 

The  Sensor  Performance  by  Sensor  Pattern  interaction  was  also  significant,  F(6,90)  = 
13.54,  p  <  .001,  making  these  main  effects  clearer.  The  interaction  appears  to  be  primarily 
driven  by  the  dissonant  data,  as  shown  in  Figure  12,  which  created  a  very  significant  effect 
on  performance  when  either  sensor  A  or  both  of  the  last  two  sensors  (B  and  C)  were 
dissonant  (OXX  vs.  XOO  in  Figure  12).  Under  these  conditions,  the  Intelligence  Officer 
correctly  identified  the  aircraft  less  than  60%  of  the  time.  This  finding  is  due  in  part  to  the 
strategy  adopted  by  most  of  the  Intelligence  Officers.  They  appeared  to  rely  primarily  on 
sensor  A  as  they  were  told  in  the  task  description  that  this  sensor  was  the  most  reliable 
(75%  vs.  50%).  The  effect  of  missing  data  in  these  same  positions  was  negligible, 
however. 
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Figure  12.  Correctness  of  Categorization  by  Sensor  Performance  and 

Pattern 


Dissonant  Data 

We  conducted  a  further  analysis  on  the  classification  performance  involving  target  sets 
with  dissonant  sensor  data  to  see  if  the  Intelligence  Officers’  decision  making  was  effected 
by  the  sensor’s  display  of  Aircraft  Type  (fighter,  bomber,  transport)  or  Identity  of  the 
targets  (enemy,  friendly,  or  a  combination  of  both).  A  16x4x3x3  (team  by  Sensor 
Pattern  by  Aircraft  Type  by  Aircraft  Identity)  ANOVA  was  performed.  This  analysis  also 
showed  the  result  that  the  two  dissonant  data  Sensor  Patterns  shown  in  Figure  12  led  the 
Intelligence  Officers  to  misclassify  aircraft  more  often  than  other  data  patterns,  F(3,45)  = 
9.64,  p  <  .001.  The  same  two  teams  that  had  the  lowest  percentage  of  correct 
classifications  overall  also  had  the  lowest  percentage  of  targets  with  dissonant  sensor  data 
classified  correctly,  F(15,1007)  =  28.58,p  <  .001. 

The  Identity  of  the  targets  did  not  effect  classification  performance.  The  analysis  of 
Aircraft  Type  revealed  that  bombers  were  misclassified  slightly  more  often  than  fighter  or 
transport  aircraft,  F(2,30)  =  5.22,  p  =  .011,  as  shown  in  Figure  13.  The  time  it  took  to 
classify  fighters  was  the  longest,  F(2,30)  =  9.39,  p  =  .001,  as  shown  in  Figure  14.  It  most 
likely  took  longer  for  classification  of  fighters  as  they  are  the  farthest  from  the  send  button. 
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Figure  13.  Categorization  Performance  by  Target  Type 
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Fighter  Bomber  Transport 

Figure  14.  Mean  Categorization  Time  by  Target  Type 


Intelligence  Officers  also  took  more  time  classifying  targets  with  dissonant  sensor  data 
when  the  aircraft  involved  were  friendly  aircraft  than  when  they  involved  enemy  aircraft  or 
mixed  presentations,  F(2,30)  =  5.67,  p  =  .008,  as  shown  in  Figure  15.  Overall,  these  data 
show  that  dissonant  data  is  more  of  a  problem  when  the  most  reliable  sensor  is  effected  or 
more  than  one  sensor  is  effected,  when  friendly  aircraft  are  involved,  and  for  certain  target 
types  more  so  than  others. 
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Figure  15.  Mean  Categorization  Time  by  Target  Identification 


20 


DISCUSSION 


While  the  microworld  task  created  for  this  test  was  somewhat  simplified  in  nature,  it  was 
successful  in  demonstrating  that  when  members  of  a  team  are  dependent  on  each  other  for 
successful  performance,  the  presence  of  both  a  Shared  Mental  Model  and  Shared  Display 
help  improve  team  performance.  Not  only  did  teams  accumulate  more  penalty  points  when 
they  did  not  have  a  Shared  Mental  Model,  but  the  Intelligence  Officer  needed  more  time  to 
send  the  requested  target  information,  thus  leaving  the  Air  Commander  with  less  time  to 
attack  and  process  incoming  aircraft.  Clearly  the  presence  of  Shared  Mental  Models, 
developed  through  an  understanding  of  the  other  team  member’s  tasks  and  goals,  was 
shown  to  be  beneficial  in  this  task. 

The  mechanism  whereby  the  Shared  Displays  aided  performance  was  not  direct  as 
expected,  however.  While  teams  were  initially  slower  when  first  given  a  Shared  Display, 
we  believe  this  is  due  to  two  factors;  they  were  spending  extra  time  looking  at  the  other 
person’s  display,  and  they  may  have  used  more  time  to  develop  a  mental  model  of  the 
other  person’s  task.  The  residual  effect  of  the  mental  model  developed  through  the  Shared 
Displays  could  be  seen  in  the  second  block  of  trials,  after  the  Shared  Display  was  removed, 
with  significantly  lower  penalty  points  and  faster  classification  times.  Shared  Displays 
therefore  appeared  to  have  a  direct  affect  of  slowing  performance,  due  to  the  demands  of 
processing  information  in  this  time  critical  task,  but  had  an  indirect  effect  of  contributing  to 
the  development  of  a  Shared  Mental  Model  thus  aiding  performance  in  later  trials. 

The  combination  of  Non-Shared  Displays  and  No  Mental  Model  was  highly  detrimental  to 
performance.  Teams  who  experienced  this  condition  first  were  unable  to  ever  develop  very 
good  performance.  Most  likely  they  developed  poor  task  strategies  that  were  never 
corrected  later-on  when  Shared  Displays  were  available.  Teams  that  had  either  Shared 
Mental  Models  or  the  Shared  Display  first  were  able  to  develop  fairly  effective 
performance  strategies.  The  best  performance  came  from  a  combination  of  Shared 
Displays  and  Shared  Mental  Models  followed  by  removal  of  the  Shared  Displays  for 
minimization  of  distractions. 

The  analysis  of  the  dissonant  sensor  data  revealed  that  it  is  handled  quite  differently  from 
missing  sensor  data,  although  the  underlying  states  of  the  world  may  be  quite  similar.  In 
other  words,  a  case  of  F16,  F16,  F18  may  actually  be  the  same  as  the  case  of  F16,  F16,  no 
data.  The  only  difference  is  whether  the  dissonant  sensor’s  data  is  displayed  or  not.  Yet, 
the  participants  in  this  study  handled  these  cases  quite  differently.  Missing  data  was  treated 
as  if  the  errant  sensor  would  have  provided  the  same  reading  as  the  other  sensors. 
Dissonant  data,  however,  significantly  effected  performance,  particularly  if  it  occurred  on 
the  primary  sensor  or  on  more  than  one  sensor. 

While  the  majority  of  the  Intelligence  Officers  adopted  the  strategy  of  relying  primarily  on 
sensor  A  to  determine  aircraft  identity,  this  strategy  did  not  always  return  a  correct 
classification.  For  example,  the  three  sensor  displays  might  show  F-16,  F-18,  F18.  In 
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some  instances,  the  aircraft  was  actually  an  F-16  and  in  others  an  F-18.  Therefore,  if  they 
relied  on  sensor  A  they  would  be  correct  half  of  the  time.  This  is  what  the  data  revealed 
(Figure  12).  However,  if  sensor  A’s  information  was  missing,  the  classification  was 
handled  very  differently.  In  these  cases,  the  Intelligence  officer  went  with  the  majority  of 
the  sensors  and  assumed  it  was  the  identity  provided  by  sensors  B  and  C.  They  did  not 
appear  to  consider  the  fact  that  sensor  A  may  report  a  different  aircraft.  There  appeared  to 
be  no  loss  in  confidence  by  the  Intelligence  Officer  under  these  cases,  as  shown  by  no 
effect  on  viewing  time. 

Even  if  a  team  did  poorly  at  classifying  targets  with  dissonant  sensor  data,  (relying  on  the 
strategy  of  always  classifying  based  on  sensor  A),  they  could  still  do  well  overall  as  only 
30%  of  the  targets  in  this  test  involved  dissonant  sensor  data.  However,  the  best  team  in 
the  study  (in  terms  of  combined  reward  minus  penalty  points)  also  had  the  highest 
percentage  of  correctly  classified  dissonant  aircraft,  showing  that  developing  effective 
strategies  for  dealing  with  dissonant  information  is  important  for  performance. 

The  detailed  analysis  of  the  dissonant  data  revealed  that  bombers  were  misclassified  more 
often  than  fighters  or  transport  aircraft.  We  are  unsure  as  to  why  this  occurred.  The  fact 
that  transport  aircraft  had  the  fastest  classification  times  may  simply  be  an  artifact  of  the 
fact  that  these  categories  were  closest  to  the  Send  Information  button  on  the  Intelligence 
Officer’s  screen  (see  Figure  3). 

Overall,  we  found  that  the  Theatre  Defense  task  was  most  difficult  for  the  Intelligence 
Officer.  Overall  team  performance  hinged  on  the  Intelligence  Officer’s  ability  to  correctly 
classify  the  aircraft  in  a  timely  fashion.  In  fact,  five  of  the  six  highest  scoring  teams  (see 
Figure  11)  adopted  a  strategy  that  relied  primarily  on  this  team  member.  In  the  teams 
using  this  strategy,  the  Intelligence  Officer  sent  over  all  aircraft  classifications  as  soon  as 
the  targets  appeared  on  the  screen,  basically  ignoring  the  Air  Commander’s  requests.  In 
the  Shared  Mental  Model  condition,  these  teams  had  the  Air  Commander  send  over  only 
those  targets  that  had  to  be  immediately  classified,  such  as  those  on  a  collision  path.  In 
the  Non-shared  Mental  Model  condition,  the  Intelligence  Officer  sent  over  most  targets 
before  the  Air  Commander  even  requested  them  and  hence  the  Air  Commander  literally 
gave  up  requesting  target  identifications.  While  this  strategy  did  produce  the  highest 
overall  points,  it  was  extremely  taxing  for  the  Intelligence  Officer  (high  workload)  and 
very  mundane  (boring  and  low  workload)  for  the  Air  Commander.  If  more  targets  (the 
maximum  in  this  study  was  six  at  a  time)  were  to  appear  on  the  screen  at  the  same  time, 
however,  this  strategy  would  break  down  as  the  Intelligence  Officer  would  be  completely 
overloaded. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


In  conclusion  this  study  supported  the  hypothesis  that  effective  team  performance  could  be 
enhanced  by  providing  teams  in  an  information  based  task  with  sufficient  information  to 
build  a  shared  mental  model  of  each  other’s  tasks  and  goals.  These  shared  mental  models 
could  be  developed  through  either  sharing  of  task  related  information  or  through  the 
provision  of  shared  displays.  Contrary  to  expectations,  the  use  of  shared  displays  did  not 
directly  aid  performance.  In  some  ways,  it  hindered  performance  in  this  task  due  to  the 
extra  time  required  to  process  the  information.  This  result  may  be  different  in  a  less  time 
pressured  task,  however,  and  should  be  researched  further. 

This  study  also  supplied  information  on  the  effects  of  dissonant  and  missing  data  on 
decision  making.  These  results  should  be  viewed  with  considerable  concern,  as  dissonant 
and  missing  data  may  often  be  present  in  battlefield  situations.  Participants  in  this  study 
appeared  to  ignore  the  potential  consequences  when  a  sensor’s  data  was  not  shown, 
treating  it  as  if  its  output  were  the  same  as  the  other  sensors.  If  that  sensor  showed  another 
value  that  was  different  than  the  other  sensors,  however,  this  had  a  considerable  effect  on 
classification  performance,  particularly  if  a  high  reliability  sensor  or  multiple  sensors  were 
involved. 

It  is  possible  that  the  participants  in  this  task,  not  being  actual  military  officers,  may  not  be 
representative  of  what  would  actually  happen  in  the  field.  Recent  research  on  a  similar 
task  with  experienced  aircraft  pilots,  however,  found  that  dissonant  data  significantly 
effected  the  probability  of  attacking  a  target  identified  by  the  primary  sensor  as  an  enemy 
when  the  dissonant  data  showed  a  friendly  aircraft  (seeking  to  avoid  fratricide),  but  not 
when  it  showed  another  enemy  aircraft  (Banbury,  Selcon,  Endsley,  Gorton,  &  Tatlock, 
1998).  These  experienced  military  pilots  also  treated  missing  sensor  data  as  if  it  were  in 
accord  with  the  classification  provided  by  the  primary  sensor.  Thus,  they  too  were 
insensitive  to  the  fact  that  the  missing  data  was  equally  likely  to  indicate  a  friendly  as  an 
enemy  in  accordance  with  the  primary  sensor  classification.  This  result  was  present  across 
various  levels  of  sensor  reliability. 

These  findings  would  tend  to  confirm  the  findings  of  the  present  study,  indicating  that 
treatment  of  missing  and  dissonant  data  is  a  significant  problem.  More  research  is  needed 
to  find  methods  of  improving  performance  under  such  conditions.  As  it  is  unlikely  that 
sensor  performance  will  always  be  perfect,  the  effects  of  conflicting  or  missing  data  on 
decision  behavior  should  be  carefully  considered  when  making  design  decisions  regarding 
which  data  to  present. 

Overall,  the  development  of  team  situation  awareness  was  shown  to  enhance  team 
performance.  The  findings  of  this  study  need  to  be  further  explored  in  relation  to  more 
complex  tasks  involving  multiple  teams  as  well  as  single  teams.  Relatively  little  is 
currently  know  about  the  effects  of  various  display  and  organizational  factors  on  the 
development  and  support  of  team  situation  awareness  in  complex  tasks  that  involve 
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battlefield  stresses  and  conditions.  More  research  is  needed  to  extend  these  findings  to 
more  realistic  conditions. 


25 


REFERENCES 


Banbury,  S.,  Selcon,  S.,  Endsley,  M.,  Gorton,  T.,  &  Tatlock,  K.  (1998).  Being  certain 
about  uncertainty:  How  the  representation  of  system  reliability  affects  pilot  decision 
making.  In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  42nd  Annual 
Meeting  (pp.  36-41).  Santa  Monica,  CA:  Human  Factors  and  Ergonomics  Society. 

Department  of  the  Air  Force  (1995).  Air  Force  executive  guidance.  Washington,  D.C.: 
Author. 

Endsley,  M.  R.  (1995).  Toward  a  theory  of  situation  awareness.  Human  Factors,  37(1),  32- 
64. 

Endsley,  M.  R.,  &  Jones,  W.  M.  (1997).  Situation  awareness,  information  dominance,  and 
information  warfare  (AIVCF-TR- 1997-0156).  Wright-Patterson  AFB,  OH:  United 
States  Air  Force  Armstrong  Laboratory. 

Kaber,  D.  B.,  &  Endsley,  M.  R.  (1997).  The  combined  effect  of  level  of  automation  and 
adaptive  automation  on  human  performance  with  complex,  dynamic  control  systems. 

In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society  41stArmual  Meeting 
(pp.  205-209).  Santa  Monica,  CA:  Human  Factors  and  Ergonomics  Society. 


26 


Appendix  A  :  Intelligence  Officer  Job  Description 


Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making 
occurs  in  the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this 
century,  future  battles  will  have  individuals  working  together  that  are  separated  by  great 
distances,  such  as  continents.  Thus,  the  transfer  and  understanding  of  information  between 
teams  will  be  critical  to  mission  success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by 
individuals  during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  Air 
Commander  or  Intelligence  officer.  Your  battle  ground  is  the  air  space  over  your  home 
base.  You  will  be  working  together  to  defeat  the  enemy.  It  is  the  air  commander’s  job  to 
ensure  that  the  home  base  is  protected  from  enemy  aircraft.  The  Air  Commander  will  be 
viewing  these  aircraft  on  a  display  similar  to  a  radar  screen.  The  Intelligence  officer's  job 
entails  providing  the  air  commander  with  information  on  these  same  aircraft  that  are 
detected  by  the  sensors.  Your  success  will  depend  upon  how  well  the  overall  team 
performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one 
of  these  workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a 
training  period  followed  by  5  trials  of  approximately  10  minutes  each.  You  may  request  a 
break  at  any  time  between  trials  and  you  will  be  given  a  15  minute  break  during  the  middle 
of  the  test.  This  will  be  followed  by  a  second  test  period  of  5  more  trials.  If  you  have  any 
questions  regarding  the  following  task  instructions  please  ask  them  before  the  testing 
begins.  You  will  get  to  practice  before  testing  begins  and  may  also  ask  questions  during 
practice. 

Intelligence  Officer 

The  United  States  has  just  learned  that  Ikestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for 
possible  attacks.  You  are  stationed  at  an  air  base  in  Western  Europe.  A  few  hours  after  the 
initial  attack  you  are  instructed  to  report  to  your  post  as  enemy  aircraft  are  making  their 
way  towards  a  U.S.  base  in  N.  Africa.  Aircraft  have  been  sent  to  intercept  and  destroy 
these  incoming  planes.  It  is  your  job  to  provide  the  air  commander  at  this  N.  African  base 
with  target  identification  information  on  these  incoming  aircraft. 

You  will  be  providing  requested  target  information.  You  will  be  shown  a  list  of  aircraft 
target  numbers  that  are  within  this  base’s  airspace.  It  is  your  job  to  decide  on  target 
identifications  and  to  send  this  information  to  the  air  commander. 

You  may  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly 
bombers,  enemy  bombers,  friendly  transports,  and  enemy  transports.  There  are  a  total  of 
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18  different  kinds  of  aircraft  that  are  in  use  by  friendly  and  enemy  forces  as  shown  in  the 
table  below. 


Plane  Categories 

Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Friendly  Bombers 

B-52 

B-1 

B-2 

Friendly  Transports 

C-130J 

C-21 

KC-135 

Enemy  Fighters 

Mig-29 

Su-35 

Su-37 

Enemy  Bombers 

TU-22M 

Tu-168 

Enemy  Transports 

An-124 

An-225 

Your  station  contains  a  set  of  head  phones  and  the  information  screen  that  is  shown  on  the 
following  page.  Targets  whose  ID  are  requested  are  shown  in  the  box  at  the  bottom  of  the 
screen  labeled  ‘Requested  Information’  in  order  of  priority.  You  need  to  monitor  this 
section  of  your  screen  as  the  target  numbers  will  be  continually  changing  as  targets  land, 
are  destroyed,  or  appear  in  your  air  space. 

You  will  have  access  to  sensor  data  about  these  targets.  The  screen  will  display  each 
sensors  best  assessment  of  the  target’s  ID,  which  may  not  always  be  correct.  It  is  up  to  you 
to  determine  the  true  aircraft  identification,  so  use  your  best  judgment.  You  can  access 
sensor  data  for  a  target  by  clicking  on  the  ‘View  Information’  button  that  corresponds  to 
the  target  number  or  by  pressing  the  number  on  the  keyboard  that  corresponds  to  the  button 
number  on  the  screen.  However,  you  may  only  see  sensor  data  for  one  target  at  a  time. 

The  sensors  are  labeled  ‘Source  A’,  ‘Source  B’,  and  ‘Source  C’.  Sensor  A  is  reliable  75% 
of  the  time.  Sensor  B  is  correct  50%  of  the  time  and  Sensor  C  is  correct  50%  of  the  time. 
Remember  to  use  these  reliabilities  when  determining  target  identification.  Be  aware  that 
not  all  sensors  may  report  information  on  a  particular  target.  For  instance.  Sensors  B  and 
C  may  have  information  on  the  first  target  requested,  but  not  sensor  A.  Thus,  you  may 
have  to  make  your  decision  on  partial  information. 

Once  you  have  viewed  the  sensor  data,  you  need  to  indicate  the  aircraft’s  identification  by 
selecting  the  aircraft  type  on  the  middle  part  of  your  screen.  All  the  possible  aircraft  are 


28 


listed  there.  After  you  have  selected  an  ID,  click  on  the  ‘Send  Information’  button  to  send 
this  information  to  the  air  commander. 

The  right  side  of  your  information  screen  'will  provide  you  with  information  regarding  the 
outcome  of  the  battle.  It  will  display  the  aircraft  num^r  and  its  actual  ID  as  well  as  the 
final  result  for  the  plane;  destroyed,  got  through,  or  collided.  This  feedback  will  give  you 
information  regarding  your  classification  of  the  aircraft.  If  you  misclassified  a  plane,  the 
“Classified  As”  and  “Actual  Type”  box  will  display  two  different  airplanes  and  the  text 
will  appear  in  red.  If  you  correctly  classified  an  aircraft,  the  text  will  appear  in  blue.  If  the 
plane  was  not  classified  prior  to  resolution,  it  will  appear  in  green  text.  Use  this 
information  to  help  you  better  identify  the  aircraft.  There  will  be  a  delay  between  making 
your  ID  and  receiving  this  feedback,  as  true  ID  and  disposition  cannot  be  determined  until 
the  plane  has  either  landed  or  been  destroyed. 

Sensor  data  for  aircraft  will  continually  appear  on  your  information  screen  until  the  testing 
time  is  up.  Thus,  you  could  potentially  process  hundreds  of  planes  in  the  10  minutes 
allotted  for  task.  Please  be  aware  that  this  task  moves  quickly.  We  want  you  to  work  as 
quickly  and  as  accurately  as  you  can. 
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Appendix  B:  Air  Commander  Job  Description 


Overview 

In  this  study  we  are  interested  in  understanding  how  individual  and  team  decision  making 
occurs  in  the  warfare  environment.  Unlike  the  wars  fought  in  the  earlier  part  of  this 
century,  future  battles  will  have  individuals  working  together  that  are  separated  by  great 
distances,  such  as  continents.  Thus,  the  transfer  and  understanding  of  information  between 
teams  will  be  critical  to  mission  success. 

During  this  task  you  will  be  asked  to  make  decisions  that  are  similar  to  ones  faced  by 
individuals  during  a  military  operation.  You  will  be  assigned  to  either  the  position  of  air 
commander  or  Intelligence  officer.  Your  battle  ground  is  the  air  space  over  your  home 
base.  You  will  be  working  together  to  defeat  the  enemy.  It  is  the  air  conunander’s  job  to 
ensure  that  the  home  base  is  protected  from  enemy  aircraft.  The  air  conunander  will  be 
viewing  these  aircraft  on  a  display  similar  to  a  radar  screen.  The  Intelligence  officer‘s  job 
entails  providing  the  air  commander  with  information  on  these  same  aircraft  that  are 
detected  by  the  sensors.  Your  success  will  depend  upon  how  well  the  overall  team 
performs. 

This  task  is  hosted  on  two  separate  computer  workstations.  You  will  be  asked  to  sit  at  one 
of  these  workstations  while  wearing  a  set  of  headphones.  We  will  first  begin  with  a 
training  period  followed  by  5  trials  of  approximately  10  minutes  each.  You  may  request  a 
break  at  any  time  between  trials  and  you  will  be  given  a  15  minute  break  during  the  middle 
of  the  test.  This  will  be  followed  by  a  second  test  period  of  5  more  trials.  If  you  have  any 
questions  regarding  the  following  task  instructions  please  ask  them  before  the  testing 
begins.  You  will  get  to  practice  before  testing  begins  and  may  also  ask  questions  during 
practice. 

Air  Commander 

The  United  States  has  just  learned  that  Dcestan  attacked  a  U.S.  base  in  the  middle  east.  The 
president  has  ordered  all  military  personnel  in  the  surrounding  areas  to  be  on  alert  for 
possible  attacks.  You  are  stationed  at  an  air  base  in  Tenya,  N.  Africa,  which  has  been  put 
on  alert.  A  few  hours  after  the  initial  attack  you  are  instructed  to  report  to  your  post  as 
enemy  aircraft  are  making  their  way  towards  your  base.  Aircraft  have  been  sent  to 
intercept  and  destroy  these  incoming  planes.  It  is  your  job  to  protect  your  base  from 
enemy  aircraft  strikes  while  allowing  your  planes  to  land  safely. 

Your  success  is  determined  by  the  number  of  reward  and  penalty  points  you  accrue.  You 
will  want  to  maximize  your  reward  points  and  minimize  your  penalty  points.  In  order  to 
do  this,  you  will  need  to  determine  which  aircraft  are  friendly  and  which  are  enemy  and 
what  type  of  aircraft  they  are.  At  the  same  time,  you  will  need  to  prioritize  incoming 
targets  based  upon  their  range  from  your  home  base  and  speed  to  determine  which  aircraft 
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are  the  most  critical  at  any  one  time.  You  will  request  aircraft  identity  information  from 
the  Intelligence  officer  (who  is  at  an  airbase  in  Western  Europe)  to  support  this 
prioritization  decision. 

Point  assignments  have  been  made  for  each  type  of  aircraft  representing  the  reward  points 
for  destroying  the  aircraft  and  the  penalty  points  for  allowing  the  aircraft  to  land  at  your 
home  base.  The  points  are  based  on  the  mission  relevance  and  lethality  of  the  aircraft.  For 
example,  transports  can  carry  many  personnel  and  therefore  could  present  a  great  threat  if 
they  land  at  a  U.S.  base,  while  the  loss  of  a  U.S.  transport  would  be  a  devastating  blow  to 
our  forces.  Friendly  aircraft  have  a  zero  penalty  for  getting  through  to  home  base  (you 
would  want  them  to  safely  land  at  home)  and  a  negative  reward  associated  with  destroying 
it  (you  would  not  want  to  destroy  your  own  planes).  Enemy  aircraft,  however,  have  both 
positive  reward  and  penalty  points. 

In  addition,  it  is  possible  for  some  targets  to  collide  with  one  another.  If  two  friendly 
aircraft  collide,  no  points  are  accrued  for  this.  If  two  enemy  aircraft  collide  the  highest 
reward  points  of  the  two  aircraft  are  given.  However,  if  a  friendly  and  an  enemy  aircraft 
collide,  you  receive  double  the  penalty  points  of  the  enemy  aircraft.  Therefore,  it  is  in  your 
best  interest  to  destroy  enemy  aircraft  before  they  collide  with  friendly  planes. 

You  will  see  6  different  types  of  aircraft:  friendly  fighters,  enemy  fighters,  friendly 
bombers,  enemy  bombers,  friendly  transports,  and  enemy  transports.  There  are  a  total  of 
18  different  kinds  of  aircraft  that  are  in  use  by  friendly  and  enemy  forces.  Below  is  a  table 
of  these  aircraft,  their  color  appearance  on  your  radar  screen  once  identified,  and  their 
associated  reward  and  penalty  points. 


Plane  Categories 

Friendly  Fighters 

Types 

F/A-18 

F-15E 

F-16 

Color 

Blue 

Blue 

Blue 

Reward 

Points 

-20 

-40 

-60 

Penalty  Points 

0 

0 

0 

Friendly  Bombers 

B-52 

Green 

-50 

0 

B-1 

Green 

-80 

0 

B-2 

Green 

-100 

0 

Friendly  Transports 

C-130J 

Turquoise 

-120 

0 

1  C-21 

Turquoise 

-140 

0 

1  KC-135 

Turquoise 

-150 

0 

Enemy  Fighters 

Mig-29 

Red 

Su-35 

Red 

80 

I  Su-37 

Red 

100 

10 

Enemy  Bombers 

TU-22M 

Orange 

10 

50 

Tu-168 

Orange 

20 

60 

Enemy  Transports  | 

An- 124 

Yellow 

50 

60 

An-225 

Yellow 

50 

60 
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You  will  view  the  aircraft  on  a  radar  screen,  similar  to  that  shown  on  the  following  page. 
Your  home  base  is  at  the  center  of  the  screen.  The  aircraft  will  appear  from  outside  the 
radar  and  travel  inward  towards  your  home  base.  It  is  your  job  to  let  the  friendly  aircraft 
land  at  the  base,  but  the  enemy  aircraft  must  be  destroyed  before  reaching  the  base.  An 
aircraft  will  get  through  if  left  alone  (it  will  land  on  its  own).  To  destroy  an  aircraft,  use  the 
computer  mouse  to  click  on  it  once.  This  gives  the  command  to  launch  missile  defense 
resources  against  the  target.  The  target  will  turn  a  light  violet  color  to  indicate  that  it  has 
been  targeted. 

Initially,  all  aircraft  appear  as  white  squares.  Below  each  square  is  the  aircraft  target 
number  and  its  speed.  In  order  for  you  to  determine  what  type  of  aircraft  is  on  your  radar, 
you  must  request  this  information  from  the  Intelligence  officer.  This  is  done  by  typing  the 
aircraft’s  number  in  the  request  box  in  the  lower  left  comer  of  the  radar  screen.  All  you 
need  to  do  is  type  in  each  number  followed  by  the  return  key.  The  Intelligence  officer  will 
receive  and  process  your  information.  It’s  identity  will  be  shown  by  the  color  of  the  square 
on  your  radar  screen  (ex.  blue  for  friendly  fighters).  The  aircraft  number  will  remain  the 
same,  but  the  penalty  points  for  the  identification  that  has  been  made  will  appear  below  the 
target  number.  This  may  or  may  not  be  correct  depending  on  the  accuracy  of  the  ID.  You 
must  decide  what  to  do  with  each  aircraft,  either  processing  it  (destroying  it )  or  letting  it 
land  at  the  base.  At  all  times  during  this  task,  your  total  reward  and  penalty  points  are 
shown  at  the  top  left  comer  of  your  radar  screen. 

Aircraft  will  continually  appear  on  your  radar  screen  until  the  testing  time  is  up.  Thus,  you 
could  potentially  process  hundreds  of  planes  in  the  10  minutes  allotted  for  each  session. 
Please  be  aware  that  this  task  moves  quickly.  We  want  you  to  work  as  quickly  and  as 
accurately  as  you  can.  These  instmctions  may  seem  a  bit  confusing,  so  we  will  start  with 
several  training  sessions  to  help  clarify  your  task. 
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Appendix  C:  ANOVA  Results 


AIR  COMANDER  DATA 


Penalty  Points 

DEP  VAR:  PPOINTS  N:  158  MULTIPLE  R:  0.501  SQUARED  MULTIPLE  R:  0.251 


ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF  MEAN-SQUARE 

F-RATIO 

P 

MENTAL  MODEL 

44.844 

1 

44.844 

9.525 

0.002 

SHARED  DISPLAY 

66.161 

3 

22.054 

4.685 

0.004 

BLOCK 

19.941 

4 

4.985 

1.059 

0.380 

SHARED  DISPLAY*MENTAL  MODEL 

14.169 

3 

4.723 

1.003 

0.394 

SHARED  DISPLAY*BLOCK 

43.099 

12 

3.592 

0.763 

0.687 

MENTAL  MODEL*BLOCK 

12.982 

4 

3.246 

0.689 

0.601 

ERROR 

612.014 

130 

4.708 

POST-HOC  ANALYSIS 

TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

44.844 

1 

44.844 

13.817 

0.021 

ERROR 

12.982 

4 

3.246 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

66.161 

3 

22.054 

6.140 

0.009 

ERROR 

43.099 

12 

3.592 

Reward  Points 


DEP  VAR:  RPOINTS  N:  158  MULTIPLE  R:  0.352  SQUARED  MULTIPLE  R:  0.124 

ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

MENTAL  MODEL 

33.624 

1 

33.624 

1.201 

0.275 

SHARED  DISPLAY 

54.952 

3 

18.317 

0.654 

0.582 

BLOCK 

50.064 

4 

12.516 

0.447 

0.774 

SHARED  DISPLAY*MENTAL  MODEL 

89.768 

3 

29.923 

1.069 

0.365 

SHARED  DISPLAY*BLOCK 

192.786 

12 

16.065 

0.574 

0.860 

MENTAL  MODEL*BLOCK 

83.721 

4 

20.930 

0.747 

0.561 

ERROR 

3640.215 

130 

28.002 
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POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED;  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

ss 

DF 

MS 

F  P 

HYPOTHESIS 

33.624 

1 

33.624 

1.606  0.274 

ERROR 

83.721 

4 

20,930 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

ss 

DF 

MS  F 

p 

HYPOTHESIS 

54.952 

3 

18.317  1.140 

0.372 

ERROR 

192.786 

12 

16.065 

Time  to  Landing 

DEP  VAR:  TIMEEXP  N:  155  MULTIPLE  R:  0.642  SQUARED  MULTIPLE  R:  0.413 

ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

MENTAL  MODEL 

127.472 

1 

127.472 

5.043 

0.026 

SHARED  DISPLAY 

1613.447 

3 

537.816 

21.277 

0.000 

BLOCK 

71.741 

4 

17.935 

0.710 

0.587 

SHARED  DISPLAY*MENTAL  MODEL 

159.188 

■  3 

53.063 

2.099 

0.104 

SHARED  DISPLAY*BLOCK 

130.047 

12 

10.837 

0.429 

0.949 

MENTAL  MODEL*BLOCK 

48.630 

4 

12.157 

0.481 

0.750 

ERROR 

3210.135 

127 

25.277 

POST-HOC  ANALYSIS 

TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

127.472 

1 

127.472 

10.485 

0.032 

ERROR 

48.630 

4 

12.157 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

1613.447 

3 

537.816 

49.627 

0.000 

ERROR 

130.047 

12 

10.837 
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Time  to  Collision 


DEPVAR:TIMECOLL  N:  153  MULTIPLE  R:  0.502  SQUARED  MULTIPLE  R:  0.252 

ANALYSIS  OF  VARIANCE 


SOURCE  SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO  P 


MENTAL  MODEL 

120.532 

1 

120.532 

3.667 

0.058 

SHARED  DISPLAY 

597.270 

3 

199.090 

6.056 

0.001 

BLOCK 

251.047 

4 

62.762 

1.909 

0.113 

SHARED  DISPLAY*MENTAL  MODEL 

52.465 

3 

17.488 

0.532 

0.661 

SHARED  DISPLAY*BLOCK 

289.357 

12 

24.113 

0.734 

0.716 

MENTAL  MODEL*BLOCK 

78.293 

4 

19.573 

0.595 

0.667 

ERROR 

4109.015 

125 

32.872 

POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 

SOURCE  SS  DF  MS  F  P 

HYPOTHESIS  120.532  1  120.532  6.158  0.068 

ERROR  78.293  4  19.573 


TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 
TEST  OF  HYPOTHESIS 

SOURCE  SS  DF  MS  F  P 

HYPOTHESIS  597.270  3  199.090  8.257  0.003 

ERROR  289.357  12  24.113 


Time  to  Attack 


DEP  VAR:  TIMEATT  N:  158  MULTIPLE  R:  0.547  SQUARED  MULTIPLE  R:  0.299 


ANALYSIS  OF  VARIANCE 

SOURCE  SUM-OF-SQUARES 


MENTAL  MODEL  16.832 

SHARED  DISPLAY  10.561 

BLOCK  0.150 

SHARED  DISPLAY*MENTAL  MODEL  5.732 

SHARED  DISPLAY*BLOCK  3.155 

MENTAL  MODEL*BLOCK  0.742 


87.921 


DF  MEAN-SQUARE  F-RATIO  P 


1 

16.832 

24.887 

0.000 

3 

3.520 

5.205 

0.002 

4 

0.038 

0.056 

0.994 

3 

1.911 

2.825 

0.041 

12 

0.263 

0.389 

0.966 

4 

0.186 

0.274 

0.894 

130 

0.676 

ERROR 


POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

16.832 

1 

16.832 

90.729 

0.001 

ERROR 

0.742 

4 

0.186 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS  F 

P 

HYPOTHESIS 

10.561 

3 

3.520  13.392 

0.000 

ERROR 

3.155 

12 

0.263 

Intelligence  Officer  Data 

Mean  Time  from  View  to  Send 


DEP  VAR: VIEWTIME  N:  159  MULTIPLE  R:  0.379  SQUARED  MULTIPLE  R:  0. 144 
ANALYSIS  OF  VARIANCE 


SOURCE  SUM-OF-SQUARES  DF  MEAN>SQUARE  F-RATIO  P 


MENTAL  MODEL 

0.950 

1 

0.950 

1.205 

0.274 

SHARED  DISPLAY 

1.223 

3 

0.408 

0.517 

0.671 

BLOCKS 

1.815 

4 

0.454 

0.575 

0.681 

SHARED  DISPLAY*MENTAL  MODEL 

1.270 

3 

0.423 

0.537 

0.658 

SHARED  DISPLAY*BLOCKS 

9.494 

12 

0.791 

1.003 

0.450 

MENTAL  MODEL*BLOCKS 

2.430 

4 

0.608 

0.770 

0.546 

ERROR 

103.322 

131 

0.789 

POST-HOC  ANALYSIS 

TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

0.950 

1 

0.950 

1.564 

0.279 

ERROR 

2.430 

4 

0.608 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

1.223 

3 

0.408 

0.515 

0.679 

ERROR 

9.494 

12 

0.791 
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Mean  Time  from  Requested  to  Send 


DEP  VARiREQUESTT  N:  156  MULTIPLE  R:  0.466  SQUARED  MULTIPLE  R:  0.217 
ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

MENTAL  MODEL 

182.174 

1 

182.174 

10.841 

0.001 

SHARED  DISPLAY 

131.326 

3 

43.775 

2.605 

0.055 

BLOCKS 

48.955 

4 

12.239 

0.728 

0.574 

SHARED  DISPLAY*MENTAL  MODEL  156.449 

3 

52.150 

3.103 

0.029 

SHARED  DISPLAY*BLOCKS 

50.633 

12 

4.219 

0.251 

0.995 

MENTAL  MODEL*BLOCKS 

13.389 

4 

3.347 

0.199 

0.938 

ERROR 

2150.863 

128 

16.804 

POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

182.174 

1 

182.174 

54.427 

0.002 

ERROR 

13.389 

4 

3.347 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F  P 

HYPOTHESIS 

131.326 

3 

43.775 

10.375  0.001 

ERROR 

50.633 

12 

4.219 

Requested  Targets 

DEP  VAR:  REQUEST  N:  160  MULTIPLE  R:  0.431  SQUARED  MULTIPLE  R:  0.186 
ANALYSIS  OF  VARIANCE 

SOURCE  SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO  P 


MENTAL  MODEL 

0.000 

1 

0.000 

0.003 

0.955 

SHARED  DISPLAY 

1.309 

3 

0.436 

5.852 

0.001 

BLOCKS 

0.017 

4 

0.004 

0.059 

0.994 

SHARED  DISPLAY*MENTAL  MODEL 

0.838 

3 

0.279 

3.747 

0.013 

SHARED  DISPLAY*BLOCKS 

0.039 

12 

0.003 

0.043 

1.000 

MENTAL  MODEL*BLOCKS 

0.039 

4 

0.010 

0.132 

0.970 

ERROR 

9.845 

132 

0.075 
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POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED;  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS  F  P 

HYPOTHESIS 

0.000 

1 

0.000  0.024  0.885 

ERROR 

0.039 

4 

0.010 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS  F  P 

HYPOTHESIS 

1.309 

3 

0.436  135.289  0.000 

ERROR 

0.039 

12 

0.003 

Targets  Classified  from  the  Top  of  the  Request  List 

DEPVAR:  TOP  N:  160  MULTIPLE  R:  0.427  SQUARED  MULTIPLE  R:  0.182 

ANALYSIS  OF  VARIANCE 


SOURCE  SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO  P 


MENTAL  MODEL 

0.477 

1 

0.477 

5.566 

0.020 

SHARED  DISPLAY 

0.394 

3 

0.131 

1.532 

0.209 

BLOCKS 

0.074 

4 

0.019 

0.216 

0.929 

SHARED  DISPLAY*MENTAL  MODEL 

1.519 

3 

0.506 

5.902 

0.001 

SHARED  DISPLAY*BLOCK 

0.048 

12 

0.004 

0.047 

1.000 

MENTAL  MODEL*BLOCKS 

0.010 

4 

0.002 

0.028 

0.998 

ERROR 

11.320 

132 

0,086 

POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  MENTAL  MODEL 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F  P 

HYPOTHESIS 

0.477 

1 

0.477 

199.288 

0.000 

ERROR 

0.010 

4 

0.002 

TEST  FOR  EFFECT  CALLED:  SHARED  DISPLAY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

0.394 

3 

0.131 

32.613 

0.000 

ERROR 

0.048 

12 

0.004 
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Target  Information  View  Time 


DEPVAR:  VIEWTIM  N: 
ANALYSIS  OF  VARIANCE 

SOURCE 

192  MULTIPLE  R:  0.728  SQUARED  MULTIPLE  R:  0.530 

SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO 

P 

TEAM 

2488.740  15 

165.916 

1.060 

0.404 

COVERAGE 

305.081  2 

152.541 

0.975 

0.381 

SENSOR 

474.968  3 

158.323 

1.012 

0.391 

TEAM*SENSOR 

7006.098  45 

155.691 

0.995 

0.496 

TEAM*COVERAGE 

4679.487  30 

155.983 

0.997 

0.484 

COVERAGE*SENSOR 

936.454  6 

156.076 

0.998 

0.432 

ERROR 

14081.700  90 

156.463 

POST-HOC  ANALYSIS 

TEST  FOR  EFFECT  CALLED:  COVERAGE 
TEST  OF  HYPOTHESIS 

SOURCE 

ss 

DF 

MS 

F 

P 

HYPOTHESIS 

ERROR 

305.081 

4679.487 

2 

30 

152.541 

155.983 

0.978 

0.388 

TEST  FOR  EFFECT  CALLED:  SENSOR 
TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

ERROR 

474.968 

7006.098 

3 

45 

158.323 

155.691 

1.017 

0.394 

Correct  Classifications 


DEPVAR:  CORRECT 

N:  192  MULTIPLE  R:  0.901  SQUARED  MULTIPLE  R:  0.81 1 

ANALYSIS  OF  VARIANCE 

SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

TEAM 

4.820 

15 

0.321 

7.758 

0.000 

COVERAGE 

3.863 

2 

1.931 

46.623 

0.000 

SENSOR 

1.884 

3 

0.628 

15.156 

0.000 

TEAM*SENSOR 

1.826 

45 

0.041 

0.980 

0.520 

TEAM*COVERAGE 

0.240 

30 

0.008 

0.193 

1.000 

COVERAGE  *SENSOR 

3.366 

6 

0.561 

13.544 

0.000 

ERROR 

3.728 

90 

0.041 

POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  COVERAGE 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

3.863 

2 

1.931 

241.905 

0.000 

ERROR 

0.240 

30 

0.008 
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TEST  FOR  EFFECT  CALLED:  SENSOR 
TEST  OF  HYPOTHESIS 


SOURCE 

ss 

DF 

MS 

F 

P 

HYPOTHESIS 

1.884 

3 

0.628 

15.470 

0.000 

ERROR 

1.826 

45 

0.041 

Dissonant  Data  Analysis 


Prioritizer  Target  Information  View  Time 


DEP  VAR: VIEWTIME  N:  1144  MULTIPLE  R:  0.181  SQUARED  MULTIPLE  R:  0.033 
ANALYSIS  OF  VARIANCE 


SOURCE  SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO  P 


SENSOR  PATTERN 

1.600 

3 

0.533 

1.237 

0.295 

ID 

3.545 

2 

1.772 

4.111 

0.017 

CATEGORY 

3.763 

2 

1.881  " 

4.364 

0.013 

SENSOR  PATTERN*ID 

0.705 

6 

0.117 

0.273 

0.950 

ID*CATEGORY 

1.310 

4 

0.328 

0.760 

0.551 

SENSOR  PATTERN  ^CATEGORY 

3.226 

6 

0.538 

1.247 

0.279 

SENSOR  PAnERN*ID*CATEGORY 

2.072 

12 

0.173 

0.400 

0.964 

ERROR 

477.657 

1108 

0.431 

Correct  Classifications  (no  team) 

DEP  VAR:  CORRECT  N:  1151  MULTIPLE  R:  0.481  SQUARED  MULTIPLE  R:  0.231 


ANALYSIS  OF  VARIANCE 


SOURCE 

SUM-OF-SQUARES 

DF 

MEAN-SQUARE 

F-RATIO 

P 

SENSOR  PATTERN 

76.463 

3 

25.488 

105.248 

0.000 

ID 

0.803 

2 

0.401 

1.658 

0.191 

CATEGORY 

0.817 

2 

0.409 

1.687 

0.185 

SENSOR  PAnERN*ID 

1.266 

6 

0.211 

0.871 

0.515 

ID*CATEGORY 

0.447 

4 

0.112 

0.461 

0.764 

SENSOR  PAnERN*CATEGORY 

0.547 

6 

0.091 

0.376 

0.894 

SENSOR  PATTERN*ID*CATEGORY 

0.956 

12 

0.080 

0.329 

.984 

ERROR 

270.016 

1115 

0.242 
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Correct  Classifications  (with  team  analysis) 

DEP  VAR:  CORRECT  N:  1151  MULTIPLE  R:  0.850  SQUARED  MULTIPLE  R:  0.722 

ANALYS!S  OF  VARIANCE 


SOURCE 

TEAM 

SENSOR  PATTERN 
ID 

CATEGORY 

SENSOR  PATrERN*ID 

SENSOR  PATTCRN^CATEGORY 

SENSOR  PATTERN*TEAM 

ID*CATEGORY 

ID*TCAM 

CATEGORY*TEAM 

ERROR 


SUM-OF-SQUARES  DF 


41.559 

15 

76.449 

3 

0.807 

2 

0.809 

2 

1.259 

6 

0.554 

6 

118.993 

45 

0.440 

4 

10.407 

30 

2.323 

30 

97.614 

1007 

MEAN-SQUARE 

F-RATIO 

P 

2.771 

28.582 

0.000 

25.483 

262.888 

0.000 

0.404 

4.164 

0.016 

0.405 

4.173 

0.016 

0.210 

2.165 

0.044 

0.092 

0.953 

0.456 

2.644 

27.279 

0.000 

0.110 

1.134 

0.339 

0.347 

3.579 

0.000 

0.077 

0.799 

0.771 

0.097 


POST-HOC  ANALYSIS 


TEST  FOR  EFFECT  CALLED:  SENSOR  PATTERN 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

76.449 

3 

25.483 

9.637 

0.000 

ERROR 

118.993 

45 

2.644 

TEST  FOR  EFFECT  CALLED:  ID 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

p 

HYPOTHESIS 

0.807 

2 

0.404 

1.164 

0.326 

ERROR 

10.407 

30 

0.347 

TEST  FOR  EFFECT  CALLED:  CATEGORY 

TEST  OF  HYPOTHESIS 

SOURCE 

SS 

DF 

MS 

F 

P 

HYPOTHESIS 

0.809 

2 

0.405 

5.224 

0.011 

ERROR 

2.323 

30 

0.077 
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Priorltizer  Target  Information  View  Time 

DEP  VARrVIEWTIME  N:  1144  MULTIPLE  R:  0.659  SQUARED  MULTIPLE  R:  0.435 

ANALYSIS  OF  VARIANCE 


SOURCE 


SUM-OF-SQUARES  DF  MEAN-SQUARE  F-RATIO  P 


TEAM 

SENSOR  PATTERN 
ID 

CATEGORY 

SENSOR  PATTERN*ID 

SENSOR  PATrERN*CATEGORY 

SENSOR  PATTERN*TEAM 

1D*CATEG0RY 

ID*TEAM 

CA1EGORY*TEAM 

ERROR 


171.337 

15 

1.772 

3 

3.762 

2 

3.779 

2 

0.659 

6 

3.258 

6 

13.327 

45 

1.332 

4 

9.954 

30 

6.036 

30 

279.183 

1000 

11.422 

40.914 

0.591 

2.115 

1.881 

6.738 

1.889 

6.768 

0.110 

0.393 

0.543 

1.945 

0.296 

1.061 

0.333 

1.193 

0.332 

1.188 

0.201 

0.721 

0.279 

POST-HOC  ANALYSIS 

TEST  FOR  EFFECT  CALLED:  SENSOR  PATTERN 
TEST  OF  HYPOTHESIS 


SOURCE 

ss 

DF 

MS 

F 

P 

HYPOTHESIS 

ERROR 

1.772 

13.327 

3 

45 

0.591 

0.296 

1.994 

0.128 

TEST  FOR  EFFECT  CALLED:  ID 

TEST  OF  HYPOTHESIS 

SOURCE 

SS  DF 

MS 

F 

P 

HYPOTHESIS 

ERROR 

3.762  2 

9.954  30 

1.881 

0.332 

5.669 

0.008 

TEST  FOR  EFFECT  CALLED:  CATEGORY 
TEST  OF  HYPOTHESIS 


SOURCE 

SS 

DF 

MS 

F 

p 

HYPOTHESIS 

3.779 

2 

1.889 

9.390 

0.001 

ERROR 

6.036 

30 

0.201 

0.000 

0.097 

0.001 

0.001 

0.884 

0.071 

0.366 

0.312 

0.224 

0.865 
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ANOVA 

ID 

SA 


GLOSSARY 


Analysis  of  Variance 
Information  Dominance 
Situation  Awareness 
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